(1) Field of the Invention
This invention relates to digital imagery and, more particularly, to automatic color balancing of digital images.
(2) Description of Related Art
Of the many concerns facing the design of a digital camera, the problem of color balancing is of primary importance. One might naively believe that the best way to resolve this issue is to create sensors that sample a spectral power distribution with extreme precision, creating an image that most accurately represents the color properties of the world. However, this is not the case when one is making cameras for human observers. A major problem lies in the fact that the human visual system does not “see” colors by simply calculating the precise wavelengths absorbed by the three photoreceptors (the Red, Green, and Blue (RGB) receptors). Rather, the brain sees colors in terms of the relative spectral distribution over the entire scene. That is, the human eye automatically adjusts to color temperature changes within the 2800 K (tungsten) to 5500 K (daylight) range through a well-known process called color constancy. As a result, a piece of paper looks white when viewed under a tungsten, halogen, or fluorescent lamp or when viewed outdoors. These conditions vary greatly in terms of the spectral power distribution ultimately reflected to the eye, yet the paper looks white, the letters look black, and the colors do not seem to change. There are, of course, limits in extreme low lighting conditions such as during and after sunsets. For example, a piece of paper that appears orange in the summer when the sun is a large red disk in the horizon has a bluish hue when the sun is setting under cloud cover. Despite the eye's shortcomings under extreme lighting conditions its color balancing properties are quite amazing. The exact mechanism by which the eye performs its color balancing function remains a mystery. However, this visual correction mechanism does not extend to images on a computer display or on photographic print. Simply taking raw Red, Green, and Blue (RGB) color channel data from a camera and dumping it to the screen will not produce an acceptable image. For instance, a native image captured in a tungsten-lit room will have an overwhelmingly yellowish tint and a digital camera must determine a way to color-balance such input data automatically. Put simply, the goal of color constancy is to change the appearance of the scene illuminant to one of constant spectral power density (i.e. white light). However, with only three spectral samples (e.g. Red, Green, and Blue) per pixel, a perfect reconstruction is impossible. Nevertheless, people have developed empirical methods that partially solve this problem.
The following equation mathematically expresses the basic process of picture taking:R(E1,E2,E3,)=(S1,S2,S3)*(reflectance)*(SPD of light source),  (1)where R is the total response, the E's are the relative responses of the light sensors (to be converted to the RGB values that the camera outputs), the S's are the relative sensitivities of these sensors to various wavelengths of light, the reflectance is a property of the object being viewed (e.g. its material—wood, metal, etc.), and SPD is the spectral power distribution of the light source. The combined product of reflectance and SPD gives the actual spectral power distribution that impinges the eyes, or that the sensors of a camera absorb. The problem is that there are many possible ways of attaining the same product, and is difficult to determine the unique reflectance of an object (which determines its perceived color).
Two major theories provide a mechanism for partially solving the problems associated with the above equation. One is called the Gray-World theory that assumes that since all colors in a given image should average to some shade of gray, one can scale the responses with respect to how much of that shade is present at the various sampled points in the visual field. Statistically, the Gray-World assumption makes perfect sense: in an image with similar dynamic ranges for each of its color components, which is generally true of the pictures taken by a camera, each spectral sample has an equal probability of being above or below a certain fixed value. By summing across hundreds of thousands of samples, one would expect the average to converge to that fixed value (shade of gray).
A camera can exploit this observation to color-balance an image. First, the camera calculates average values for the red, green, and blue components from raw image data. By the Gray-World assumption, these three components should have the same value. However, if the illuminant does not have a constant SPD, these average values will deviate from one another in a way that is characteristic of the color appearance of the illuminant. One, in essence, factors out the average assumed to represent the SPD of the light source in order to attain the relative reflectance of the objects as if viewed under a white light. For instance, data obtained under a tungsten illuminant will have higher red and green averages than blue averages. To force the Gray-World assumption to hold, the camera can scale each of the color channels independently to equalize the RGB averages. The camera begins by picking an overall value for the averages to converge. One convenient value is simply the average of the three component averages. Multiplying each color channel by the overall average divided by the component average will yield an image for which the Gray-World assumption is true. This approach is not exact, but it manages to scale the responses in terms of surrounding conditions, and is therefore a good approximation for color balancing.
Another such approach that also tries to approximate the lighting conditions is know as the Perfect-Reflector hypothesis that assumes that the brightest object in an image reflects the best approximation of the SPD of the light source. Therefore, the RGB values of the brightest or most reflective spot are used to scale the RGB's of all other objects in the image in order to obtain a relative reading for the entire image. Thus, if one wishes to reconstruct the color appearance of the subject under white light, one should scale the image's RGB components to render the specular white. The scale factor for each color channel is just 255/specular value for an 8-bit system. This algorithm focuses only on the specular or glossy surfaces in the image and does not take into account the overall appearance. In fact, this may be a better approach for images where there is too little color variation for the Gray-World assumption to hold.
There are other algorithms for color constancy in the literature. In E. H. Land et al., titled “Lightness and Retinex theory,” Journal of Optical Society of America, Vol. 61, no. 1, PP. 1-11, January 1971, the authors adopted an approach based on illuminant spatial frequency in order to compensate for variations in color. A computational model was built for modeling color constancy by L. T. Mahoney et al. and reported in the article “A Computational Model for Color Constancy,” Journal of Optical Society of America, Vol. 1, PP. 29-33, 1986, that attempted to model surface reflectance by low-dimensional families of surfaces. This model was later simplified using linear models taught by D. H. Marimont et al. and reported in the article “Linear Model for Color Constancy,” Journal of Optical Society of America, Vol. 1, PP. 1905-1913, 1992. In an article by D. H. Brained et al, titled “Bayseian Color Constancy,” Journal of Optical Society of America, vol. 14, PP. 1393-1411, 1997, the authors proposed modeling color constancy as a Bayesian model that uses the cues such as surface reflectance and illuminant spectra to make Bayesian decisions that maximize expected utility. B. V. Funt et al., as demonstrated in their article titled “Is machine color constancy good enough?” in Proc. of ECCV, PP. 445-459, 1998, found this approach inaccurate. An attempt to improve the Perfect-Reflector algorithm was made by D. A. Forsyth and was described in the article titled “Sampling, Resampling and Color Constancy,” in Proc. of CVPR, PP 300-305, 1999. The attempt proposed a resampling approach as a solution, but resulted in a method that is computationally slow and that required several frames to accurately correct for variations in color.
There are also several patented approaches for achieving color constancy, broadly classified into hardware-based and software-based approaches. The software-based approaches typically provide a solution based on modifying a camera sensor in some fashion to achieve color constancy. U.S. Pat. No. 6,417,863 disclosed numerous embodiments of a method for adjusting the color balance of a display. In one embodiment, the color balance of a display is adjusted by modulating two selected colors so that at some point during the modulation the two colors appear substantially the same on the display and the modulation adjusts to change the point at which the colors appear substantially the same. The color balance of the display is modified, in part, based on a measurement of the modulation adjustment. In another embodiment of a method for adjusting the color balance of a display, the reference teaches the modification of the luminance and/or chromaticity of a color from the color space of the display, and modulation of luminance and/or chromaticity of a color from a reference color space, where, over time the applied modulation results in the two colors becoming substantially the same. Thus, the modulation adjusts so that the colors appear substantially the same at a selected point during the modulation of both colors, and the color balance of the display modifies partly based on a measurement of the modulation adjustment. This approach is not automatic in its balancing because it requires the user to pre-select, either the colors or provide reference color display for modulation adjustment. Accordingly, it is less attractive due to the need for user input.
In U.S. Pat. No. 6,388,648, a liquid crystal display (LCD) projection unit employs a luminance and color balance system employing an LCD array characterization lookup table storing multiple sets of luminance and gamma correction values that are user-selectable to control luminance and color balance. The lookup table values are determined by measuring the transfer function of the LCD array in the projection unit for each of a set of red (“R”), green (“G”), and blue (“B”) input data values, converting the resulting S-curve responses (color responses) to corresponding sets of gamma responses and scaling the gamma responses to generate multiple R, G, and B families of luminance and gamma corrected values. Color balance is adjusted by selecting the particular R, G, and B families of luminance and gamma corrected values that cause the LCD projection unit to match a predetermined ratio of R, G, and B luminance values. Luminance adjusts by selecting sets of families of R, G, and B luminance values that maintain the color balance ratio while causing the LCD to transmit a desired overall luminance. This technique has the disadvantage of requiring the storage of a large lookup table in order to tackle variations in lighting. Furthermore, the luminance and gamma correction values have to be user-selected which implies that the approach is not automatic.
U.S. Pat. No. 5,231,504 disclosed an improved method for using a linear mixing space, such as a CIE (Commission Internationale de l'Eclairage) color space, to determine the quantities of coloring agents needed to reproduce a color, applicable, for example, to printing a color that matches an original color on a cathode ray tube (CRT) display. The calculations match hue, saturation and reflectance so that the matching color appears like the original color and so that those characteristics of an original image are preserved. An original color's definition converts into coordinates in the linear mixing space. An achromatic region in the linear mixing space, preferably elliptically shaped, includes the coordinates of the neutral coloring agents. Only quantities of neutral coloring agents match the original colors falling within the achromatic region. This approach however, does not address the issue of color constancy of scenes as imaged by a camera under varying lighting conditions. The focus is more on reproducing a print of the colors on a CRT display.
U.S. Pat. No. 5,504,524 presented a method of controlling the color balance of a video signal that comprises green, red, and blue channel signals. The patent identifies pixels that have a green intensity within a peak intensity band. For these pixels, the reference examines the red and blue intensities to identify pixel subsets within respective red and blue peak bands. If the red and/or blue peak band(s) are/is greater than the green peak band, the gain of the red and/or blue channel(s) is increased. If the red and/or blue peak band(s) are/is lower than the green peak band, red and/or blue peak band(s) are/is determined for the entire video frame, and compared with the green peak band. If the red and/or blue peak band(s) for the entire frame are/is lower than that for the green channel, the gain of the red and/or blue channel(s) are/is increased. However, this algorithm does not perform well for images where there is too little color variation or for images with high specula reflections.
In light of the current state of the art and the drawbacks to current systems mentioned above, a need exists for a system that would automatically color balance an image to correct for various lighting situations. That is, it would minimize discrepancies that exist between the natural response of a human eye and the response of an image-capturing device.